Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

don't fail CI matrices fast #1804

Merged
merged 1 commit into from
May 18, 2023
Merged

don't fail CI matrices fast #1804

merged 1 commit into from
May 18, 2023

Conversation

pmeier
Copy link
Member

@pmeier pmeier commented May 15, 2023

Reference Issues or PRs

It is really annoying to see a large matrix completely cancelled just because one job failed. For example: https://github.com/nebari-dev/nebari/actions/runs/4981394220/jobs/8915592409. This is especially true, if the individual parameters aren't strongly correlated. For example, if tests fail on Python 3.8 it is likely that they also fail on 3.9. In contrast, if the provider test fails on AWS, this gives very little insight if the same would also fail on GCP or the like.

Thus, unless we have very strict budget concerns here, it would improve DevX quite a bit to not fail the matrix builds fast. The infracost CI has fail-fast: true set, but there is no comment in the PR that added it (#1316) as for why.

What does this implement/fix?

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds a feature)
  • Breaking change (fix or feature that would cause existing features not to work as expected)
  • Documentation Update
  • Code style update (formatting, renaming)
  • Refactoring (no functional changes, no API changes)
  • Build related changes
  • Other (please describe):

@Adam-D-Lewis Adam-D-Lewis self-requested a review May 18, 2023 16:10
@Adam-D-Lewis
Copy link
Member

It is really annoying to see a large matrix completely cancelled just because one job failed. For example: https://github.com/nebari-dev/nebari/actions/runs/4981394220/jobs/8915592409. This is especially true, if the individual parameters aren't strongly correlated. For example, if tests fail on Python 3.8 it is likely that they also fail on 3.9. In contrast, if the provider test fails on AWS, this gives very little insight if the same would also fail on GCP or the like.

I agree with this

@Adam-D-Lewis Adam-D-Lewis merged commit 82510c0 into nebari-dev:develop May 18, 2023
@pmeier pmeier deleted the dont-fail-fast branch May 18, 2023 16:27
@iameskild iameskild mentioned this pull request May 18, 2023
Merged
10 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Adam-D-Lewis Adam-D-Lewis Adam-D-Lewis approved these changes

Assignees
No one assigned
Labels
area: CI/CD 👷🏽‍♀️ area:developer-experience 👩🏻‍💻 needs: review 👀 This PR is complete and ready for reviewing
Projects
🪴 Nebari Project Management
Archived in project
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@pmeier @Adam-D-Lewis